feat: Support OFED with DTK #711

rollandf · 2023-12-19T10:47:15Z

In Openshift, in order to OFED containter to be able to download and compile the needed Kernel files, it is required to install a cluster-wide entitlement.

This requirement is not user friendly.

In order to avoid this, a container image with the needed files is available in Openshift distributions.
This image is called DriverToolKit aka DTK.

By using this container as a side-car to MOFED container, the modules can be compiled without entitlement.

Changes:
API:

DTK is 'true' by default, and can be changed by env
variable in the Operator Deployment

OFED state:

In case of OCP and 'useOcpDriverToolkit' is true, find DTK image based on NFD label of node.
If available, add to MOFED DS a DTK container, change entrypoint logic.

rollandf · 2023-12-19T15:10:45Z

Note that the API with MOFED container is not finalized yet.

api/v1alpha1/nicclusterpolicy_types.go

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml

vasrem · 2023-12-20T11:09:25Z

pkg/state/state_ofed_test.go

+}
+
+func (d *openShiftClusterProvider) GetClusterType() clustertype.Type {
+	return clustertype.Kubernetes


nit: switch to clusterType.Openshift to be consistent with what the methods below return.

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml

pkg/state/state_ofed.go

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml

rollandf · 2023-12-25T08:31:07Z

/retest-blackduck_scan

adrianchiris · 2023-12-25T11:58:13Z

deployment/network-operator/README.md

@@ -447,6 +447,7 @@ containerResources:
 | `ofedDriver.upgradePolicy.waitForCompletion.podSelector`    | string | not set                           | specifies a label selector for the pods to wait for completion before starting the driver upgrade                                                                             |
 | `ofedDriver.upgradePolicy.waitForCompletion.timeoutSeconds` | int    | not set                           | specify the length of time in seconds to wait before giving up for workload to finish, zero means infinite                                                                    |
 | `ofedDriver.containerResources`                             | []     | not set                           | Optional [resource requests and limits](#container-resources) for the `mofed-container` container                                                                             |
+| `ofedDriver.useOcpDriverToolkit` | bool | `true` | In OpenShift, use Driver Toolkit image to compile OFED drivers                                                                                                                                                  |


prevent setting to this to true in validation webhook if we are not in openshift ? WDYT ?

i.e if not openshift and this is true -> fail CR create/update

i now remember we discussed the flow :) so maybe we should just ignore this on non openshift

alternative: dont introduce any API changes for this in CRD, add ENV var in operator (DISABLE_DTK or similar)

since for openshift we do want dtk to always run. in special cases where we dont, we can deploy operator with this env var.

this is best option imo as going forward we either want dtk or precompiled

Removed from Helm

adrianchiris · 2023-12-25T12:00:37Z

hack/templates/values/values.template

@@ -226,6 +226,7 @@ ofedDriver:
      # podSelector: "app=myapp"
      # specify the length of time in seconds to wait before giving up for workload to finish, zero means infinite
      # timeoutSeconds: 300
+  useOcpDriverToolkit: true


do we really want to have this defined in values ?
(also i dont see we use it in the nicclusterpolicy CR template)

Removed from helm

adrianchiris · 2023-12-25T12:11:38Z

pkg/state/state_ofed.go

@@ -735,3 +753,27 @@ func (s *stateOFED) handleRepoConfig(
 	}
 	return nil
 }
+
+// getOCPDriverToolkitImage gets the DTK ImageStream and return the DTK image according to RHCOS version
+func (s *stateOFED) getOCPDriverToolkitImage(ctx context.Context, rhcosVersion string) (string, error) {


rhcosVersion -> ostreeVersion

this exists also for rhel (OSTREE_VERSION) i assume.
and we should support dtk on rhel as well right ?

OSTREE_VERSION="412.86.202301311551-0"

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml

adrianchiris · 2023-12-25T12:59:15Z

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml

+          {{- end }}
+          {{- end }}
+          volumeMounts:
+            {{- if.AdditionalVolumeMounts.VolumeMounts }}


do we need additional mounts for DTK ?

rollandf · 2024-01-03T22:45:09Z

/retest-nic_operator_kind

main.go

deployment/network-operator/templates/operator.yaml

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml

In Openshift, in order to OFED containter to be able to download and compile the needed Kernel files, it is required to install a cluster-wide entitlement. This requirement is not user friendly. In order to avoid this, a container image with the needed files is available in Openshift distributions. This image is called DriverToolKit aka DTK. By using this container as a side-car to MOFED container, the modules can be compiled without entitlement. Changes: API: - DTK is 'true' by default, and can be changed by env variable in the Operator Deployment OFED state: - In case of OCP and 'useOcpDriverToolkit' is true, find DTK image based on NFD label of node. - If available, add to MOFED DS a DTK container, change entrypoint logic. Signed-off-by: Fred Rolland <[email protected]>

adrianchiris

lgtm

rollandf · 2024-01-07T11:31:49Z

/retest-nic_operator_kind

rollandf · 2024-01-08T12:08:14Z

@e0ne @adrianchiris Can we have this in the beta?

rollandf added the on hold This enhancement is currently on hold pending additional clarification and evaluation label Dec 19, 2023

rollandf force-pushed the dtk-new branch from eaa0b6a to b25431c Compare December 19, 2023 10:48

rollandf force-pushed the dtk-new branch 2 times, most recently from 2c6b188 to 52ed808 Compare December 20, 2023 10:05

vasrem reviewed Dec 20, 2023

View reviewed changes

rollandf force-pushed the dtk-new branch 4 times, most recently from 864d53f to a274a04 Compare December 25, 2023 06:48

adrianchiris reviewed Dec 25, 2023

View reviewed changes

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml Show resolved Hide resolved

adrianchiris reviewed Dec 25, 2023

View reviewed changes

rollandf force-pushed the dtk-new branch from a274a04 to e8e8f6c Compare December 25, 2023 13:41

rollandf force-pushed the dtk-new branch 3 times, most recently from 2bc57ff to e21665e Compare January 3, 2024 22:28

rollandf added on hold This enhancement is currently on hold pending additional clarification and evaluation and removed on hold This enhancement is currently on hold pending additional clarification and evaluation labels Jan 4, 2024

adrianchiris reviewed Jan 7, 2024

View reviewed changes

main.go Outdated Show resolved Hide resolved

deployment/network-operator/templates/operator.yaml Show resolved Hide resolved

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml Show resolved Hide resolved

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml Show resolved Hide resolved

adrianchiris reviewed Jan 7, 2024

View reviewed changes

manifests/state-ofed-driver/0050_ofed-driver-ds.yaml Show resolved Hide resolved

rollandf force-pushed the dtk-new branch from e21665e to 3f29c9b Compare January 7, 2024 10:31

adrianchiris approved these changes Jan 7, 2024

View reviewed changes

rollandf removed the on hold This enhancement is currently on hold pending additional clarification and evaluation label Jan 8, 2024

e0ne approved these changes Jan 8, 2024

View reviewed changes

e0ne merged commit 9f03156 into Mellanox:master Jan 8, 2024
16 checks passed

rollandf deleted the dtk-new branch February 28, 2024 08:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support OFED with DTK #711

feat: Support OFED with DTK #711

rollandf commented Dec 19, 2023 •

edited

Loading

rollandf commented Dec 19, 2023

vasrem Dec 20, 2023

rollandf Dec 20, 2023

rollandf commented Dec 25, 2023

adrianchiris Dec 25, 2023 •

edited

Loading

adrianchiris Dec 25, 2023

adrianchiris Dec 25, 2023

rollandf Dec 25, 2023

adrianchiris Dec 25, 2023 •

edited

Loading

rollandf Dec 25, 2023

adrianchiris Dec 25, 2023

rollandf Dec 25, 2023

adrianchiris Dec 25, 2023

rollandf Dec 25, 2023

rollandf commented Jan 3, 2024

adrianchiris left a comment

rollandf commented Jan 7, 2024

rollandf commented Jan 8, 2024

feat: Support OFED with DTK #711

feat: Support OFED with DTK #711

Conversation

rollandf commented Dec 19, 2023 • edited Loading

rollandf commented Dec 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rollandf commented Dec 25, 2023

adrianchiris Dec 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianchiris Dec 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rollandf commented Jan 3, 2024

adrianchiris left a comment

Choose a reason for hiding this comment

rollandf commented Jan 7, 2024

rollandf commented Jan 8, 2024

rollandf commented Dec 19, 2023 •

edited

Loading

adrianchiris Dec 25, 2023 •

edited

Loading

adrianchiris Dec 25, 2023 •

edited

Loading